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(57) Abstract: Using little more than a control device such as a handheld infrared remote control and a viewing device such as 
digital "smart" television or a standard or high definition television with a set top box, a viewer is able to enjoy various pre-recorded 
or televised programming and is also able to easily access supplemental content that is "something like" the programming currently 
being viewed or any accessible data element in that content The supplemental content is sensitive to the present context of the 
programming being displayed and is also personalized for the viewer based on the viewer* s profile. The personalized supplemental 
content is "across media," meaning that it is obtained or derived from various media sources and media types, including additional 
information through broadcasted programs as well as Internet sources, but is presented to the viewer in a easy-to-use, consistent, 
complementary and uniform way. The viewer selects only those items of personalized information of interest, and the selections 
appear merged with the program on the viewing device, thereby presenting the information to the viewer in a timely and convenient 
manner. The viewer's selections are used to update the viewer's profile. A collaborative filtering database may also be used in 
various ways to supplement the viewer's profile. 
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Fusion of media for information sources. 



BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to interactive television, and more particularly to 
the integration of interactive television with various data sources. 
5 Description of Related Art 

Interactive TV merges the mass appeal of traditional TV with the rich, 
interactive power of the Internet. Rudiment forms of interactive TV are available from 
various vendors, including WebTV Networks, Inc. of Mountain View, California, which 
provides receivers for accessing MSNBC interactive television. More advanced forms of 

10 interactive TV are being developed. For example, as more fully described in Backer et al., "A 
Framework for Interactive Television Based on Internet Standards," 
http://www.microsoft.com/ atveffwpaperl .asp, April 14, 1999 (last update date), several 
companies have been collaborating over the past months on a framework for supporting 
interactive television content based largely on the existing Internet standards. This framework 

1 5 in fact builds on a number of components, including Internet Protocol (IP) as the primary 
transport for all data; Session Announcement Protocol and Session Description Protocol 
(SAP/SDP) for announcing streams of interactive content; Hypertext Markup Language 
(HTML) and related data formats as mechanisms for describing the presentation of graphics, 
text, video, and other multimedia; Uniform Resource Identifiers (URIs) as the mechanism for 

20 referencing all multimedia elements, including television broadcasts and other content 
delivered over uni-directional links; ECMAScript and the Document Object Model (/.e., 
JavaScript™ and JScript™) for dynamic control of HTML content and the synchronization 
of HTML content to broadcast video; and Multipurpose Internet Mail Extensions / Hypertext 
Transport Protocol (MIME/HTTP)-style headers as the representation for content meta-data. 

25 The authors believe that these six elements provide virtually all of the services necessary to 
design and deliver compelling interactive television content, and that only a few elements of 
innovation are required. A specific framework related to this effort is expressed in the 
Advanced Television Enhancement Forum Specification, Draft Version 1.1 r26, February 2, 
1999. 
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While advances are being made in producing and delivering television 
enhancements, further enhancements are needed that use existing standards and a plethora of 
new component technologies to provide the viewer of television programming with greater 
access to a variety of different content types appropriate to the viewer in a convenient and 
5 intuitive way. 

Summary of the Invention 

One embodiment of the present invention is a method for displaying content 
from multiple information sources to a viewer, comprising establishing a viewer profile for 
the viewer based on various interests of the viewer; presenting a program to the viewer on a 

10 viewing device; generating items of personalized information based on the program and on 
the viewer profile; superimposing a visual selection aid over the program, the visual selection 
aid comprising index symbols for the items of personalized information, the index symbols 
on the visual selection aid being accessible to the viewer for selection; identifying a selection 
of one of the index symbols by the viewer to obtain a selected one of the items of personal 

15 information; and superimposing the selected item of personalized information over the 
program. 

Another embodiment of the present invention is a receiver apparatus for 
obtaining content from multiple information sources for viewing by a viewer, comprising a 
program detector; an program-related additional information detector; a video output coupled 

20 to the program detector for driving a viewing device to present the program to the viewer; an 
adaptive viewer profile database; an implicit filter coupled to the adaptive viewer profile 
database, the implicit filter having an input coupled to the program-related additional 
information detector for generating items of personalized information based on the program- 
related additional information and on the adaptive viewer profile database; means for 

25 superimposing a visual selection aid over the program, the visual selection aid comprising 
index symbols for the items of personalized information and the index symbols on the visual 
selection aid being accessible to the viewer for selection; means for identifying a selection of 
one of the index symbols by the viewer to obtain a selected one of the items of personal 
information; and means for superimposing the selected item of personalized information over 

30 the program. 

Yet another embodiment of the present invention is a computer program 
product comprising a computer readable medium having program logic recorded thereon for 
enabling a computer-enabled apparatus to display content from multiple information sources 
to a viewer on a viewing device, comprising: means for establishing a viewer profile for the 
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viewer based on various interests of the viewer; means for presenting a program to the viewer 
on the viewing device; means for generating items of personalized information based on the 
program and on the viewer profile; means for superimposing a visual selection aid over the 
program, the visual selection aid comprising index symbols for the items of personalized 
5 information, the index symbols on the visual selection aid being accessible to the viewer for 
selection; means for identifying a selection of one of the index symbols by the viewer to 
obtain a selected one of the items of personal information; and means for superimposing the 
selected item of personalized information over the program. 
Brief Description of the Drawings 

10 Figure 1 through Figure 6 are pictorial representations of various viewing 

screen displays over time, in accordance with the present invention. 

Figure 7 is a block schematic diagram of a media fusion system, in accordance 
with the present invention. 

Figure 8 is a flowchart of an illustrative operating session of the media fusion 

1 5 system of Figure 7, in accordance with the present invention. 
Detailed Description of the Preferred Embodiment 

Using little more than a control device such as a handheld infrared remote 
control and a viewing device, which includes, for example, digital "smart" television, a 
standard or high definition television with a set top box, a personal computer, or even a 

20 personal viewing device having a small screen or viewing glasses and a wireless or wired 
Internet connection, a viewer is able to enjoy various pre-recorded or televised programming 
such as movies, sitcoms, theater, musicals, dance, sports, live action, hosted auctions, feature 
advertisements, and so forth, and is also able to view suggestions for supplemental content 
that is "something like" the programming currently being viewed or any accessible data 

25 element in that content. The supplemental content is sensitive to the present context of the 
programming being displayed and is also personalized for the viewer based on the viewer's 
profile. The personalized supplemental content is "across media," meaning that it is obtained 
or derived from various media sources and media types. The viewer may easily select any of 
the personalized supplemental content, which is then presented to the viewer on the viewing 

30 device in a consistent, complementary and uniform way. 

In a standard broadcast television context, for example, the media fusion 
system enhances a viewer's experience by suggesting items of personalized information such 
as program information, biographical information on any actors and characters appearing in 
the program, information on the music and effects used in the program, background 
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information on the making of the program, program related references, historical data on the 
program (e.g. movie data), sports statistics, interactive menus, advertising information, on- 
line shopping, other statistical or detailed information, or a combination thereof that is related 
both to the program and to the viewer's interests. The personalized information is obtained 
5 from among the plethora of information available from different media sources might interest 
the viewer. The media fusion system merges the screen presentation of the suggestions as 
well as, in some cases, the personalized information itself with the regular broadcast 
programming, thereby presenting the information to the viewer in a timely and convenient 
manner rather than requiring the viewer to access individual pieces of information as is 

10 necessary in a conventional Web-based environment. 

An illustrative example of a viewer's experience with the media fusion system 
is shown in Figures 1 through 3, which pertain to receiving program related information 
during a standard broadcast television program. Figure 1 shows a broadcast program in 
progress after activation of the media fusion system. Activation of the media fusion may be 

1 5 by manual action of the viewer or automatically. Manual activation may be in any desired 
manner, as by pressing a function button on a remote control or by moving an on-screen 
cursor to a particular location on the screen. Automatic activation may be a periodic 
activation or activation at predefined times such as the beginning and end of a program and 
during breaks in the program, for example. 

20 A cube 5 appears in the upper right corner of the screen (it may be located in 

any desired place on the screen) to indicate that the media fusion system is active and that 
three items of personalized information are made available to the viewer, as indicated by 
symbols on three facets of the cube 5. If desired, the cube 5 may be programmed to rotate 
upon command, providing up to six facets to indicate the availability of items of personalized 

25 information. Alternatively, any three dimensional shape having a multiplicity of facets may 
be used. Multiple cubes or other three dimensional shapes may be used if more facets are 
required, or older items of personalized information may be retired as newer items of 
personalized information become available. Two dimensional matrices and one dimensional 
lists may be used as an alternative to three dimensional shapes, if desired, although these are 

30 most useful as nested menus to aid the viewer in making interactive choices once an item of 
personalized information has been accessed. The symbol appearing on a facet to indicate that 
an item of personalized information is available may be an arbitrary number such as shown in 
Figures 1 through 3, or may be an icon such as a musical note, talking head, motion picture 
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camera, dollar sign, exclamation mark, and so forth that suggest the general nature of the 
item of personalized information referenced by the facet. 

The cube 5 is just one specific type of a visual selection aid. While the visual 
selection aid may be one dimensioned or multi-dimensioned, it basically functions to provide 
5 the viewer with a convenient visual clue for selecting items of personalized information. The 
items of personalized information are indexed by any suitable symbols, which are displayed 
to the viewer on the visual selection aid. 

Although the viewer may ignore the cube 5 and continue watching the 
television program, typically the viewer desires certain information or wishes to take certain 

10 actions related to the current television program. This is done by "selecting" a facet of the 
cube 5. How a selection is made depends on the type of controller used by the viewer. For 
example, a selection may be made with a system that supports an on-screen cursor by moving 
the cursor over the desired facet and pressing the select button, and selection may be made 
with a system that does not support an on-screen cursor by pressing a function button and 

1 5 then the number corresponding to the desired facet. 

Figure 2 shows the result when the viewer selects the facet labeled "1." The 
item of personalized information represented by facet "1" is program related information 
identified by the title "1. On Safari (documentary)," which appears on the screen. Because 
the viewer of Figure 2 regularly watches wildlife and conservation programming, the media 

20 fusion system has presented an item of information that pertains to other conservation 

programming. The media fusion system accomplishes this by maintaining an adaptive profile 
of the viewer. The ability of the viewer's adaptive profile to generate relevant and valuable 
recommendations and links improves with experience. 

If the viewer is not interested at the moment in more information on the "On 

25 Safari" documentary, the viewer may cancel the selection in any convenient manner, such as, 
for example, by pressing a "cancel" button on a remote control or moving a cursor over a 
"cancel" icon (not shown). The number "1" remains on a face of the cube to indicate that the 
item of personalized information is still available if the viewer later desires to access it. The 
normal advertisement stream and other ancillary content continues to be presented to the 

30 viewer. However, if the viewer is interested in more information, the viewer makes a further 
selection in any convenient manner, such as, for example, by again selecting the facet "1" or 
by selecting the displayed text. Additional program related information is displayed, 
illustratively "Documenting a team of explorers, we find them fighting for their lives while 
on safari in Kenya .... Catch this program tonight: Discovery Channel: 8 PM" as shown in 
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Figure 3. The viewer may use this additional program related information in any desired 
manner, including, for example, simply remembering to tune in at 8 PM to watch the 
program or by initiating the automatic programming of the viewer's VCR to record the 
program. Since the additional program related information is superimposed over the current 

5 broadcast program, the viewer simply cancels the selection to return to the program when her 
use of the additional program related information is complete. 

Other facets of the cube represent other types of personalized information. An 
illustrative example of a viewer's receiving advertising information with the media fusion 
system during a standard broadcast television program is shown in Figure 4 and Figure 5. As 

10 the wildlife program progresses, the viewer selects the facet of cube 5 labeled "2." Figure 4 
shows the result of this selection, which is a display on the screen of "2. Landrover 
(advertisement)." The advertisement is based on the current program content as well as the 
viewer's adaptive profile, so the advertising is more likely to be of specific value to the 
viewer. 

15 if the viewer is not interested at the moment in more information on the 

"Landrover" advertisement, the viewer may cancel the selection. The number "2" remains on 
a face of the cube to indicate that the item of personalized information is still available if the 
viewer later desires to access it. The normal advertisement stream and other ancillary content 
continues to be presented to the viewer. However, if the viewer is interested in more 

20 information, the viewer makes a further selection in any convenient manner, such as, for 
example, by again selecting the facet "2" or by selecting the displayed text. The full 
Landrover advertisement appears superimposed over the program, as shown in Figure 5. The 
full advertisement is, illustratively, "The New 4x4, the best in its category, is ready for a new 
type of owner. Ready for an owner like you . . . ." The viewer may use this advertisement in 

25 any desired manner depending on the capabilities of the viewer's set top box, ranging from 
simply taking note of it for general interest to automatically contacting an on-line vendor 
based on the displayed information to schedule an appointment for test driving the vehicle or 
even to directly purchase the vehicle. 

An illustrative technique for using, for example, the related program 

30 information shown in Figure 3 or the Landrover advertisement shown in Figure 5 as a 

gateway to other viewer activities is to enable the viewer to evoke a menu or a nested set of 
interactive menus by again selecting the appropriate facet or by selecting the displayed * 
advertisement. The menu may have a number of choices, including for the related program 
information example a "Program VCR" choice and for the Landrover advertisement example 
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an e-commerce choice. For some activities, display of the program is discontinued at least 
temporarily so that the activity screen may be displayed. 

The third item of personalized information represented by the third facet of 
cube 5 is, for example, an article on native African wildlife. This is because the viewer 
5 frequently consults on-line reference material while watching nature programming, a 
behavior captured in the viewer's adaptive profile. 

Other examples to illustrate on-line shopping, biographical information, and 
interactive programming follow. 

Example 1: On-line Shopping. A viewer is watching an advertisement for a particular model 
10 of car. The viewer is interested in additional information about the car and "selects" the 

image of the car. The media fusion system responds with information relevant to the viewer's 
interests, such as the retail price of the car in the viewer's geographical region, Web sites or 
dealerships from which the car may be purchased, safety information about the car (the 
media fusion system having "learned" that the viewer is safety-conscious from previous 
1 5 viewing selections), and a video of the car being driven on a test track (the media fusion 
system having "learned" that the viewer likes action scenes from previous viewing 
selections). The relevant information is obtained from a variety of sources, both local and 
from the Web. 

Example 2: Program Related References, On-line Shopping, and Enhanced Communications. 
20 The viewer of a movie enjoys motion picture soundtracks, a behavior captured in the 

viewer's adaptive profile along with the viewer's frequent use of e-commerce and email. 

Finding that he or. she enjoys the music, the viewer notices that one of the items of 

personalized information made available by the media fusion system is a "current audio" 

selection. The viewer selects this item, and the system responds by showing the composer, 
25 artist and title. The viewer selects "further options," and is presented with a screen menu of 

personalized options, including, for example, a music purchase option and an email option. 

These options permit the viewer to effortlessly do a number of actions common to the viewer, 

such as purchasing the music for herself, or informing friends of the music, or purchasing the 

music for a friend and informing him that it is coming. 
30 Example 3: Movie Database. The viewer is watching a movie. As the movie ends, the viewer 

is presented with various items of personalized information, including a suggestion to watch 

another movie similar to the movie just viewed. 

Example 4: Detailed Information. The viewer engages in frequent stock transactions with an 
on-line broker, a behavior captured in the viewer's adaptive profile. During a commercial 
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advertisement for a product from a particular company, the viewer is presented with various 
items of personalized information, including a financial report on the company and a link to 
the viewer's on-line broker. 

An illustrative media fusion system, which includes various computers, local 
5 area and global networks, enabling software, and databases, is shown in Figure 7. The 
viewer's portion of the media fusion system, the viewer's "receiving equipment," may be 
implement in any of a variety of ways, including contained in a separate housing such as a set 
top box, built into a broadcast television, or running as an application on a computer or 
another type of smart appliance having an attached monitor. Broadcast programming and 

10 Internet access is available by a variety of communications techniques, including radio, 
cable, satellite, and phone line. Set top boxes are presently available from a variety of 
vendors, including, for example, General Instruments, Inc. of Horsham, Pennsylvania, and 
Scientific Atlanta, Inc. of Atlanta, Georgia, and even more powerful set top boxes are in 
development and production. The set top box may be a Java console, for example. Where the 

15 media fusion system is implemented using the Internet, suitable Internet-resident computers 
include, for example, servers and clients. Suitable computers, communications networks, and 
set top boxes are well known in the art. 

On-screen results 40 (Figure 7) are displayed on a television monitor 
connected to the set top box. It will be appreciated that the use of a television monitor is 

20 illustrative, and that many other techniques are well known to persons of ordinary skill in the 
relevant arts for displaying dynamic graphical data. In this example, the on-screen results 40 
include three components, a TV program component 42, a enhanced behavior component 44, 
and a server-based content component 46. These components originate generally from data 
20. It will also be appreciated that various circuits and methods for processing dynamic 

25 graphical data, including overlaying graphical data from different sources, are well known to 
persons of ordinary skill in the relevant arts. 

The TV program component 42 originates from a primary broadcast 22, and 
typically is carried by FM transmission, satellite, or cable. The TV program component 42 
may also be received in other ways, such as over DSL or ISDN or other types of advanced 

30 digital communications techniques, and from prerecorded materials such as video tape, 
digital tape, CD-ROMs, and DVDs. Program selection is in accordance with viewer 
selections 50. 

The enhanced behavior component 44, which includes enhanced behavior 
information such as links related to the content of the primary broadcast 22 and other 
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information designed to enhance the behavior of the viewer's receiving equipment, originates 
from additional information 24. The additional information 24 is coordinated with the 
primary broadcast 22 and preferably originates with it, is context sensitive by design, and 
illustratively accompanies the primary broadcast 22 in the vertical blanking interval (for 

5 analog television), for example. The additional information 24 is filtered by an 

explicit/implicit filter 14 and the resulting enhanced behavior information 44 is superimposed 
over the TV program 42. The explicit/implicit filter 14 also acquires information about the 
televised program context from the additional information 24. 

The server based content component 46 originates from various remote data 

10 sources such as servers 28 and 29, based on requests, illustratively URL or CGI instructions, 
from the explicit/implicit filter 14. Servers 28 and 29 may reside on a local area network, on 
an enterprise network, or on the Internet, for example. In the event that servers 28 and 29 
reside on the Internet, they are compliant with Web protocols. The Internet connection allows 
many different types of information and extended connections to be provided to the viewer 

1 5 while a program is being viewed. Examples of types of information that a viewer may 

requested and that may be displayed as server based content 46 are an article about African 
wildlife from the New York Times Web site and detailed advertising information and vehicle 
specifications from the Landrover Web site. An example of a type of extended connection 
supported is a purchase option, by which a viewer can conveniently purchase an advertised 

20 product or service on-line. 

The explicit/implicit filter 14 provides filtering for information from various 
sources, so that only context-sensitive information that is within the scope of the viewer's 
profile is furnished as the enhanced behavior component 44. The performance of the 
explicit/implicit filter 14 is dependent on the viewer's personal profile, as defined by 

25 information contained in the adaptive viewer profile database 12, and on the viewer's 

selections 50. The explicit/implicit filter 14 acts on the additional information 24 by blocking 
enhanced behavior information that is not within the scope of the viewer's personal profile 
and passing enhanced behavior information that is within the scope of the viewer's personal 
profile. The explicit/implicit filter 14 also requests information from the servers 28 and 29 as 

30 required by the viewer's selections 50 and in accordance with the viewer's personal profile. 

Through implicit filtering, the adaptive viewer profile database 12 captures the 
personality of a particular viewer (or several viewers, if more than one viewer uses the set top 
box), and is progressively refined and augmented to create an adaptive profile based on the 
viewer's history together with marketing statistics. The personal profile develops over time, 
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in that the system "learns" what type of information to offer to the viewer based on various 
criteria such as what the viewer usually selects and the amount of time that the selection 
holds the viewer's interest, if relevant, and adapts accordingly. For example, if advertising 
information rather than reference material is usually selected, the system works out a ratio to 
5 reflect the viewer's behavior and uses the ratio in responding to the viewer's current 
programming choice or specific request. 

Explicit filtering involves building or supplementing a viewer's profile based 
on her answers to specific questions. For example, detecting a new viewer, the 
explicit/implicit filter 14 presents a pre-programmed list of questions to the viewer to 

10 establish an initial profile, thereby allowing the media fusion system to function on at least a 
rudimentary level initially. Explicit filtering is also useful for confirming trends identified by 
the explicit/implicit filter 14, thereby saving time in updating the viewer's profile, and for 
resolving any ambiguity that may be detected in a viewer's developed profile by posing 
questions to the viewer to resolve the ambiguity. Implicit filtering involves building a 

1 5 viewer's profile based on her selections, including television program selections, items of 
personalized information accessed, Web links accessed, Web sites explored, purchases made, 
and so forth. The personal profile of each viewer using the media fusion system is stored in 
an adaptive viewer profile database 12, which preferably resides on the set top box along 
with the explicit/implicit filter 14 and is locally accessible to the explicit/implicit filter 14. 

20 Implicit filtering may, if desired, draw from collaborative data in building a 

viewer's profile. For example, Figure 7 shows that the explicit/implicit filter 14 has a bi- 
directional path to a profile mirror 27, which is associated with a collaborative filtering 
database 26 and a server 29. The profile minor 27 may be another computer or another 
process running on the same computer as the server 29. It is not part of the collaborative 

25 filtering database 26, but rather includes profile information derived from the collaborative 
filtering database 26 and, if desired, from the adaptive viewer profile database 12. The server 
29 illustratively has "push" capabilities, which enable the server 29 to push meaningful 
information to the viewer based on information from the profile mirror 27. 

The collaborative filtering database 26 is a sizeable database of profiles of 

30 viewers who visit the Web sites hosted by the server 29. A number of these viewers are likely 
to have profiles that are similar to the profile of the viewer whose profile is maintained in the 
adaptive viewer profile database 12. These similar profiles are identified by comparing the 
viewer's profile with the collaborative filtering database 26. Differences between the 
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viewer's profile and the similar profiles in the collaborative filtering database 26 represent 
possible viewer interests, and as such are useful to the viewer in a variety of ways. 

One way in which the differences between the viewer's profile and the similar 
profiles in the collaborative filtering database 26 is useful is as part of a response to one of 
5 the viewer selections 50. In this case, the differences are furnished by the profile mirror 27 to 
the server 29 which pushes information to the viewer as server based content 46. 

Another way in which the differences between the viewer's profile and the 
similar profiles in the collaborative filtering database 26 is useful is to create additional items 
of personalized information under the assumption that the viewer might be interested in the 
1 0 same things as other viewers who have similar profiles. In this case, the differences are 
furnished by the profile mirror 27 to the explicit/implicit filter 14 which includes symbols 
that index the differences on the visual selection aid as items of personalized information. If 
the viewer selects any of them, the selection is used to update the adaptive viewer profile 
database 12. 

1 5 Another way in which the differences between the viewer's profile and the 

similar profiles in the collaborative filtering database 26 is useful is to augment the profile of 
the viewer whose profile is maintained in the adaptive viewer profile database 12, under the 
assumption that the viewer will be interested in the same things as other viewers who have 
similar profiles. The augmentation is performed in any convenient manner. For example, one 

20 technique involves copying the adaptive viewer profile database 12 to the profile mirror 27, 
updating the profile mirror with appropriate information from the collaborative filtering 
database 26, and writing the updated viewer profile back to the adaptive viewer profile 
database 12. Another technique involves copying the adaptive viewer profile database 12 to 
the profile mirror 27 so that the differences may be identified, then furnishing the differences 

25 to the explicit/implicit filter 14 which handles updating the adaptive viewer profile database 
12. The new data in the viewer's profile is used by the explicit/implicit filter 14 to generate 
items of personalized information, and is evaluated over time by the explicit/implicit filter 14 
just as is other data in the viewer's profile. 

The collaborative filtering database 26 is also useful for establishing a "starter 

30 kit," or an initial profile of data for a new user. Preferably, the new viewer is asked a number 
of explicit start up questions as part of the explicit filtering process by the explicit/implicit 
filter 14, and then the adaptive viewer profile database 12 is augmented from the 
collaborative filtering database 26. 
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Suitable algorithms and software for performing data collection and filtering 
and for maintaining databases such as the adaptive viewer profile database 12 and the 
collaborative filtering database 26 is well known to persons of ordinary skill in the art. 
Generally known as software agents, suitable software is based on any of a variety of 
5 techniques, including Bayesian probability-based recommendation models, decision tree 
models, neural network models, and distance metrics. The database itself may be organized 
in any suitable way, including flat file, relational, and object oriented. Examples of the 
Bayesian probability-based recommendation model are described in, for example, the 
following articles, which hereby are incorporated herein in their entirety by reference: 

10 Pazzani, M., Muramatsu, J., and Billsus, D., Syskill & Webert: Identifying Interesting Web 
Sites, in Proceedings of the National Conference on Artificial Intelligence, Portland, Oregon, 
1996; Billsus, D. and Pazzani, M., Learning Probabilistic User Models, in Workshop Notes 
of Machine Learning for User Modeling, Sixth International Conference on User Modeling, 
Chia Laguna, Sardinia, 1997; and Pazzani, M. and Billsus, D., Learning and Revising User 

1 5 Profiles: The Identification of Interesting Web Sites, in Machine Learning 27, 1 997, pp. 3 1 3- 
331 . An example of collaborative filtering is the technology being researched at the Media 
Laboratory of the Massachusetts Institute of Technology, Cambridge, MA and 
commercialized by such companies as Firefly Networks, Inc. of Cambridge, MA. 

To enhance the seamlessness of media fusion, the primary broadcast 22 may 

20 be provided with embedded highlights. Embedded highlights are associated with specific 
objects on the screen, with the location of those objects being furnished as part of the 
additional information 24 or in any other suitable manner. A highlight is of any suitable form, 
such as, for example, a colored region or tag that tracks with the movement of its associated 
screen object. For example, the screen object may be a sweater worn by an actor, and the tag 

25 may be a colored rectangle that is superimposed on the sweater and moves with it. The 
highlight may be selected by the viewer in any convenient manner, in which event 
information is furnished about the item and other options, such as a purchase option, are 
presented. If the tag is not selected, it fades from the screen. 

Figure 8 is a flowchart showing an operating session of the media fusion 

30 system of Figure 7. Figure 8 is merely illustrative, since the media fusion system may be 
programmed in a variety of ways depending on the experiences, skill and preferences of the 
system architect and computer programmers. When the viewer's receiver is turned on (step 
100), the viewer's receiver determines using a viewer ID and password or other such 
mechanism whether the viewer is a new viewer or a previous viewer. If the viewer is new, 
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the receiver (through the explicit/implicit filter 14 of Figure 7, which preferable is resident in 
the receiver) poses various questions to establish a start-up profile for the new viewer (step 
112). Once having established a few general traits about the new viewer, the receiver (again 
through the explicit/implicit filter 14) may, if desired, augment the new viewer profile from 
5 the remote collaborative filtering database 26. Alternatively, a new viewer may adopt another 
viewer's profile or may choose an initial profile from among a number of pre-constructed 
"standard" profiles. Having created a viewer profile, the receiver then generates and makes 
available for selection new items of personalized information (step 1 14). Depending on the 
results of the decision step 110, either a newly created viewer profile or an existing viewer 

1 0 profile is used in subsequent operation. 

As additional information 24 becomes available through the primary broadcast 
22 (step 120), the receiver updates the viewer's profile in accordance therewith (step 122) 
and generates and makes available for selection new items of personalized information (step 
124) based on the new additional information and the viewer's updated profile. Alternatively, 

15 the additional information 24 may be stored and the adaptive viewer profile database 12 
updated periodically or when the storage is nearly full. 

If the viewer makes a selection (step 1 30), the receiver responds in an 
appropriate manner. For example, if the selection is a channel selection or program change 
(step 140), the receiver changes the channel and updates the viewer's profile accordingly 

20 (step 142), and generates and makes available for selection new items of personalized 

information (step 144) based, for example, on the known characteristics of the new channel 
(news, weather, sports, general entertainment) and the viewer's updated profile. 

If the viewer's selection is of an item of personalized information or an 
embedded tag (step 150), the receiver updates the viewer's profile accordingly (step 152), 

25 and generates and makes available for selection personalized information and/or options (step 
154) based on the viewer's selection and the viewer's updated profile. A personalized option 
may itself have the characteristics of an item of personalized information, in that it may be 
selected in a subsequent execution of step 150 and acted upon in subsequent executions of 
steps 152 and 154. 

30 Personalized options may include such choices as a purchase option, a chat 

option, a comparison shopping option, and email option, a Web site browse option, and so 
forth. If the viewer's selection is of such a personalized option (step 160), the receiver 
updates the viewer's profile accordingly (step 162) and executes the option in the appropriate 
manner (step 164). 
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The various methods and apparatus described herein in some aspects enhance 
a viewer's viewing experience by providing easy access to relevant information, and in some 
aspects provide a viewer of television programming with greater access to a variety of 
different content types appropriate to the viewer in a convenient and intuitive way. 
5 The description and applications as set forth herein are illustrative and are not 

intended to limit the scope of the invention, which is defined in the following claims. 
Variations and modifications of the embodiments disclosed herein are possible, and practical 
alternatives to and equivalents of the various elements of the embodiments are known to 
those of ordinary skill in the art. These and other variations and modifications of the 
10 embodiments disclosed herein may be made without departing from the scope and spirit of 
the invention, as set forth in the following claims. 
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1 . A method of displaying content from multiple information sources to a viewer, 
comprising: 

establishing (100, 1 10, 1 12) a viewer profile for the viewer based on various 
interests of the viewer; 
5 presenting a program (22) to the viewer on a viewing device (40, 42); 

generating (1 14, 124, 144) items of personalized information based on the 
program and on the viewer profile; 

superimposing a visual selection aid (5) over the program, the visual selection 
aid comprising index symbols (1, 2, 3) for the items of personalized information, the index 
10 symbols on the visual selection aid being accessible to the viewer for selection; 

identifying (150) a selection of one of the index symbols by the viewer to 
obtain a selected one of the items of personal information; and 

superimposing the selected item of personalized information (44, 46) over the 

program. 

15 

2. A method as in claim 1 further comprising updating (152) the viewer profile in 
accordance with the viewer's selection of the selected item of personalized information. 

3. A method as in claim 1 further comprising augmenting the viewer profile in 
20 accordance with a collaborative data base (26). 

4. A method as in claim 3 wherein the personalized information generating step 
comprises: 

generating at least one of the items of personalized information (27, 29) based 
25 on user profiles in the collaborative database (26) similar to the viewer profile (12), at least 
one of the index symbols on the visual selection aid being an additional index symbol 
therefor; 

identifying a selection (50) of the index symbol for the item of personalized 
information based on the similar user profiles in the collaborative database; 
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updating the viewer profile (12, 14) in accordance with the viewer's selection 
of the item of personalized information based on the similar user profiles in the collaborative 
database; and 

superimposing the item of personalized information (46) based on the similar 
5 user profiles in the collaborative database over the program. 

5. A method as in claim 1 wherein the personalized information generating step 

comprises: 

detecting additional information (24) related to the program; 
1 0 filtering the additional information in accordance with the viewer profile (14) 

to obtain filtered additional information; and 

generating at least some of the items of personalized information (44) from the 
filtered additional information. 

15 6. A method as in claim 1 wherein the personalized information generating step 

comprises generating at least one of the items of personalized information from information 
on content available from a Web site (28, 29). 

7. A method as in claim 1 wherein the selected item of personalized information 
20 comprises a reference to a Web site, further comprising: 

identifying a selection of the Web site reference by the viewer (50); and 
displaying content from the Web site on the viewing device (28, 29, 46). 

8. A method as in claim 7 wherein the Web site comprises a push server (29) and 
25 a collaborative filtering database (26), the Web site displaying step comprising: 

identifying differences (27) between the viewer profile (12) and similar user 
profiles in the collaborative database (26); and 

displaying content pushed from the push server on the viewing device in 
accordance with the differences (29, 46). 

30 

9. A method as in claim 8 wherein the Web site comprises a profile mirror (27) 
and wherein the difference identifying step comprises: 

downloading the viewer profile from an adaptive viewer profile database (12) 
to the profile mirror (27); 
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updating the profile mirror from the collaborative filtering database (26) to 
obtain an updated profile mirror (27); and 

pushing content from the push server (29) in accordance with the updated 
profile mirror for display (46) on the viewing device. 

5 

10. A method as in claim 9 further comprising uploading the updated profile 
mirror from the profile mirror (27) to the adaptive viewer profile database (12). 

11. A receiver apparatus for obtaining content from multiple information sources 
10 (20) for viewing (40) by a viewer, comprising: 

a program detector (22); 

an program-related additional information detector (24); 

a video output coupled to the program detector for driving a viewing device 
(40) to present the program (42) to the viewer; 
1 5 an adaptive viewer profile database (12); 

an implicit filter (14) coupled to the adaptive viewer profile database, the 
implicit filter having an input coupled to the program-related additional information detector 
for generating items of personalized information based on the program-related additional 
information and on the adaptive viewer profile database; 
20 means (44) for superimposing a visual selection aid (5) over the program, the 

visual selection aid comprising index symbols (1, 2, 3) for the items of personalized 
information and the index symbols on the visual selection aid being accessible to the viewer 
for selection; 

means (50) for identifying a selection of one of the index symbols by the 
25 viewer to obtain a selected one of the items of personal information; and 

means (44, 46) for superimposing the selected item of personalized 
information over the program. 

12. A receiver apparatus as in claim 1 1 further comprising means (14) for 

30 updating the adaptive viewer profile database in accordance with the viewer's selection of the 
selected item of personalized information. 



13. 

comprises: 



A receiver apparatus as in claim 12 wherein the implicit filter further 
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means (27, 14) for generating at least one of the items of personalized 
information based on user profiles in a remote collaborative database (26) similar to the 
viewer profile, at least one of the index symbols on the visual selection aid being an 
additional index symbol therefor; and 
5 means (14, 50) for updating the viewer profile in accordance with a viewer's 

selection of the item of personalized information based on the similar user profiles in the 
collaborative database. 

14. A receiver apparatus as in claim 1 1 further comprising means (27, 14) for 

1 0 augmenting the adaptive viewer profile database in accordance with a collaborative data base 
(26). 

1 5. A computer program product comprising a computer readable medium having 
program logic recorded thereon for enabling a computer-enabled apparatus to display content 

1 5 from multiple information sources to a viewer on a viewing device, comprising: 

means (1 00, 1 1 0, 1 1 2) for establishing a viewer profile for the viewer based on 
various interests of the viewer; 

means for presenting a program (22) to the viewer on the viewing device (40, 

42); 

20 means (1 14, 124, 144) for generating items of personalized information based 

on the program and on the viewer profile; 

means for superimposing a visual selection aid (5) over the program, the visual 
selection aid comprising index symbols (1 , 2, 3) for the items of personalized information, 
the index symbols on the visual selection aid being accessible to the viewer for selection; 
25 means (150) for identifying a selection of one of the index symbols by the 

viewer to obtain a selected one of the items of personal information; and 

means for superimposing the selected item of personalized information (44, 
46) over the program. 



30 



16. A computer program product as in claim 1 5 further comprising means (1 52) 

for updating the viewer profile in accordance with the viewer's selection of the selected item 
of personalized information. 



WO 01733839 PCT/EPO 0/1 0214 

19 

1 7. A computer program product as in claim 1 5 wherein the means for generating 

personalized information comprises: 

means for generating at least one of the items of personalized information (27, 
29) based on user profiles in the collaborative database (26) similar to the viewer profile (12), 
5 at least one of the index symbols on the visual selection aid being an additional index symbol 
therefor; 

means for identifying a selection (50) of the index symbol for the item of 
personalized information based on the similar user profiles in the collaborative database; 

means for updating the viewer profile (12, 14) in accordance with the viewer's 
10 selection of the item of personalized information based on the similar user profiles in the 
collaborative database; and 

means for superimposing the item of personalized information (46) based on 
the similar user profiles in the collaborative database over the program. 



15 



18. A computer program product as in claim 1 5 further comprising means for 

augmenting the viewer profile in accordance with a collaborative data base (26). 
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